The present disclosure relates in general to data logging, and in particular to the debugging of the logging of data pertaining to the operation of a data stream processing server.
Traditional database management systems (DBMSs) execute queries in a “request-response” fashion over finite, stored data sets. For example, a traditional DBMS can receive a request to execute a query from a client, execute the query against a stored database, and return a result set to the client.
In recent years, data stream management systems (DSMSs) have been developed that can execute queries in a continuous manner over potentially unbounded, real-time data streams. For example, a typical DSMS can receive one or more data streams, register a query against the data streams, and continuously execute the query as new data appears in the streams. Since this type of query (referred to herein as a “continuous query”) is long-running, the DSMS can provide a continuous stream of updated results to a client. Due to the continuous nature of such queries, debugging or diagnosing problems within continuous queries is extremely difficult. With a complex event processing (CEP) server, continuous query language (CQL) has been used in describing the continuous queries.
Currently, diagnosing or debugging of continuous queries can be done by performing various levels of logging, such as input/output adapter, output bean, operator, store, synopsis, queues, or the processing nodes in the event processing network level. However, this method cannot provide enough simplicity and flexibility for properly debugging the continuous queries. Usually the problem of logging methods include: too much logging data to analyze, not being able to change the state and continue, not being able to trigger conditions to enable logging, etc. Furthermore, some of the debugging cannot be done using just logging, for example, the pattern operator involving complex state.
DSMSs are particularly suited for applications that require real-time or near real-time processing of streaming data, such as financial ticker analysis, physical probe/sensor monitoring, network traffic management, and the like. Many DSMSs include a server application (referred to herein as a “data stream processing server”) that is configured to perform the core tasks of receiving data streams and performing various operations (e.g., executing continuous queries) on the streams. It would be desirable to have a framework for logging data pertaining to the operation of such a data stream processing server to facilitate performance tuning, debugging, and other functions. Hence, improvements in the art are needed.